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Estimation of Effect Size 



A UNIFIED APPROACH TO THE ESTIMATION 
OF EFFECT SIZE IN META-ANALYSIS 

Introduction 

In research synthesis, statistical methods aire used to describe the findings of studies 
under re^-iew. Typically, the selected studies are considered independent experiments 
concerning the behaviors of some common dependent variables. Each study usually consists 
of a control group (Y) and at le2Lst one treatment condition fX). The results of each study 
are summarized by an index of effect size {6). The first estimate of effect size, developed 
by Cohen (1966, 1967) and modified by Glass (1976), is of the form g = (X-Y)/Sy or 
the difference between mecuis of the treatment (x) and control (F) groups divided by the 
stauidard deviation of the control group (Sy). Important contributions to the estimation 
theory of effect size are attributable to Rosenthal (1978), Hedges (1981), Rosenthal & 
Rubin (1982), Kraemer (1983) and Hedges & Olkin (1985). 

Let -Yi, Xn and Yi, Y^n represent some random samples of the treatment and 
control normcil populations; fxx and /iy, the population means of X and Y, respectively; 
and rr, the standard deviation of the response scores of all subjects in the combined treat- 
ment and control population. The present investigation addresses parameteric methods to 
estimate Cohen's effect size 6 = (^>>'^f^v) from a single experiment or for a single study in 
meta-analysis. The enumeration of all possible estimators of a would result in a countless 
number of estimators of 6. Table 1 provides a limited list of some popular estimators of 
(7-2, namely Sf and Sf for t = 1, . . 5. 



Insert Table 1 about here 

Generally, an unbiased estimator of is 5^ whereas its maximum likelihood estimator 
is 5^. The distributions of 5^^ and 5? can be summarized respectively as fS^/(T^'^x^{j) and 
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Estimation of EITect Size 

fSf/<^^^X^{fp where x^(/) and X^(/) denote chi-squares with f and / degrees of freedom, 
respectively, and "-^'^ meaning " is distributed as'\ The estimates of a and their associated 
degrees of freedom are listed in Table 1 under various conditions of variance homogeneity 
and heterogeneity. 

Under variance homogeneity (Case 1), as well as under variance heterogeneity with 
known variance ratios (Case 2), four estimat ;;s of a are listed in Table 1. These estimates 
are derived after the procedures to estimate effect sizes as (i) suggested by Glass (1976) 
(5i and 53 and their maximum likelihood estimator (MLE) versions, §1 and 53), and (ii) 
proposed by Hedges (1981) (52 and 54 and their MLE counterparts, 52 and 54). Also 
listed are the estimates of a under variance heterogeneity (Case 3) eis introduced by Welch 
(1938) and Cohen (1966) (55 and its MLE counterpart, 55). Other estimates of can be 
derived by modifying the values of and in Table 1. For example, the values below 
sure mathematically equivalent to 5| = [{n - 1)5^ + (m - l)S^]/{n + m - 2): 

Si = (E - + E (Xi - yf]/{n + m - 2), and 

5| = [nSjc + mS^]/{n + m - 2), 
where 5^ = EC^'f " - 1), 5f = E(^'t - - 1), S% = Ei^i-xf/n, and 5f = 

E(^'-Wm 

In Table 1, the degrees of freedom for the unbiased estimates of cr (/,, i = 1, 
4) are less than those for the corresponding MLE counterparts (/,). It is important to 
note that the estimators of effect size are non-central t statistics whose distributions are 
characterized by the specifiation of degrees of freedom, f and /, 

The main objective of this investigation is to examine the principal statistical prop- 
erties of the estimators of 6 under variance homogeneity (Case 1), variance heterogeneity 
with known variance ratios (Case 2) and for the Behrens- Fisher problem (Case 3). The 
derived estimators are compared according to the criteria of their magnitudes, unbiased- 
ness and mean-square errors (USE). The present investigation can be considered as an 
extension of the existing studies on the estimation of effect size in me* a- analysis from 
three perspectives: 

(i) Except for the estimators of 6 listed under Case 1 in Table 1, oiher possibilities 
have not yet been examined. 

(ii) A generalized approach in effect size estimation is undertaken such that common 
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Estimation of Effect Size 

properties of estimators of 5 axe derived. These properties can be applied to estimates 
that can be considered as mathematically equivalent to those given in Table 1. 

(iii) Although the estimation of treatment effects in the presence of variance hetero- 
geneity has been discussed (Wilcox, 1985), the estimation of effect sizes for tht Behrens- 
Fisher problem in research synthesis is addressed here for the first time. In particular, the 
bias of the estimators of effect size computed according to the methods of Case 1 (Variance 
homogeneity) in the presence of Vciriance heterogeneity is investigated. 

(iv) It will be shown that estimators of effect size are unstable, namely, variances of 
the estimators increase with effect size (6). In this study, a relatively stabilized estimator 
with respect to 6 is identified and its properties, examined. 

In the following, the common statistical features of estimates of 6 are treated first in 
general terms. Then, specific charax:teristics o" some selected estimators, which are formed 
35 functions of 5* ztnd St listed in Table 1, axe analyzed. Finally, general properties of the 
derived estimators are re-examined by means of Monte-Carlo results. 

General Properties of Estimators of Effect Size 

Model Specification and Assumptions 

The properties of the estimators of effect size are studied on the basis of the following 
three assumptions: 

(Al) The random samples X and Y axe distributed normally with meejis /xjc, /*y and 
finite variance g\ ajid cr^, respectively. The population moments in the distibutions of X 
and Y are unknown; 

(A2) X, F, 5^, and 5^ are mutually independent; and 

(A3) The standardized difference between the treatment and control effects is repre- 
sented by the effect size 6 = (/ijc - fiy)/(T where the forms of a due given in Table 1. 

It will be shown that, for i = 1, 5, the biased estimators of 6 are of the forms 
gt =.(X - Y)/St and = (X - Y)/St^ and the corresponding unbiased estimators are K = cjg^ 
and fe, = CfQt^ respectively; where c/ = (y/f / 2)rlu i)/2] ^ ^/ ~ [^///]c^. The distributions of 
g, h and h axe found to be noncentral t with the noncentral parameter defined generally 
as A = [(^/(^xJy)^^ where ^^{jf^fj = [a^ j -f cr^y] zmd a = aj^ = ay under variance homogeneity 
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or cr^ = {a^ <^y)/2 in the presence of the Behrens-Fisher problem. In addition, some linear 
transformations of g, g, h and A, such as the variance* stabilizing and shrunken estimators, 
are also considered. 

The derivation and properties of estimators of 6 aie influenced by the following two 
properties of the functions c/ and cji (i) the degrees of freedom f and / should be larger 
than 2, (ii) c/ and cire monotonically increasing functions of the degrees of freedom (0.72 < 
(c/,cy) < 1 for 3 < (/,/) < oo) (Hedges and Olkin, 1985, Table 2, p. 80), or equivalently, the 
inverse of these functions are monotonically decreasing (l.38 > (cj^, cj^) > 1 for 3 < (/, /) < oo. 
For example, if /2 > fi then c/, > c/, and cj^ < cjK 

The Distributions of the Estimators of 6 

The distribution of a generalized biased estimator of 6 is formalized a£ follows: 
Proposition i . Consider the estimators g - (X -Y)/S and g = (!x - y)/5; where S^/a^^ 
X^^y/f and is a maximum likelihood estimator of such that fi^^^^ = {flf)o^ for f,f 
The distributions of g and g are ki^j.k-^s) and fci^^jt-i*)? respectively; where k = (^(x^y)/^^ 
k = {\fj)ky t{f,ii,) is a noncentral t-distribution with degrees of freedom f and noncentral 
parameter A. 

Proof : The estimator g can be written as, 

(Ir V\ / C - ^f'x-v) ''u-v) _ (g+A) _ fc(g+A) 



where A = (mx -My)/^^(x-»r), ^ = ^(x^y)I^^ ^-i\r(0,l) and Slcr^^Jxls)! 

According to Johnson and Welch's (1939), the distribution of g is of the form as 
specified. Moreover, the relationship g = [\Jflf]g is tenable since § = - F)/5, and S = 
[\l flf\S. Hence, the distribution of g is obtained a given above. || 

Corollary 1 . (Biased estimators) 
(i) The means and variances of g and §, respectively, are: 

(a) (Exact), = cjH and ^5 = cj% a] = (//(/ - 2)){k^ + i^) _ and C7| = (//(/ - 
2))(fc^+^^)-/i|; where the expressions for c/ and c^- have been specified previously, 

(b) (Approximate). /T, = [(4/ - l)/4(/- and = [(4/- l)/4(/- 

= [//(/ - 2)W + - and = [//(/ - 2)W + - /i^ 
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(c) (Asymptotic), fig^oo = 5, and /x^,oo = /^s.oo; (^loo = (*"^ + (*V2/)), and ^ = . 

(ii) The varisince-stabilizing and normalizing transformations of g and g eire: 

= b^^$inh^^{ba'^g) = 6~^/n|6^ + o^^l, and 
ff- = 6-isinfe-^(6a-^g) = b-^ln\bg + 
where = [//(/ - 2)]P, a? = [//(/ - 2)]F, b' = [//(/ - 2)] - , 6^ = [//(/ - 2)] - cj, 
6--i[5m/i-^(6a-^g) - sm/i-^(6a-i5)] % iNr(0,l), 6-^[szn/i-^(6a-^ff) - sm/i-H6a-^5)]%i\r(0, 1), and 
denotes "is asymptotically distributed as". 

(iii) The bias and mean-squcire errors of g and g caji be derived as: 

(a) Bia${g) = [c]^ - and 3ias{g) = [cy-^ - 

(b) MSE{g) = [//(/-2)](i2+52)-(i_2c7^)52, and MSE{g) = [f/{f-2)]{k^+6^y{l-2cj^)6\ 
Proof : The proof will be carried out for the results associated with the estimator 

g. Similcir arguments can be derived for the results related to the maximum likelihood 
estimator g by means of the transformation g = [y/ f/f]g* 

(a) The well-known result Z = ^^~^K ^N{0, 1) implies that ^E=f^-xfn//. Let W = 
{X - Y)IS^_Y) unbised estimate of A = fc~^5. Then W can be rewritten as: 

^(X-Y) Jx-\± V 

Then, from Johnson and Welch (1939), fiw = rJ^A and = [/(I + A^)/{f - 2)] - /x^. 
Since implies that /x^ = fc/xvv and cr^ = fc^cr^; hence, the results follow. 

(b) The values of ilg and aie based on the approximation c) = 1 - [3/(4/- 1)] with 
an error < .0003 for /> 10 (Hedges, 1981). 

(c) The limiting distribution of W is normal with mean of A and vciriance of [1 + 
(AV2/)] (Johnson and Welch, 1939, p. 367). Since Z-J\r(0,l) and by applying the central 
limit theorem, it yields: 



Z = t(y,^) x/l + (AV2/) = k{g - 5)/v/l + (fc-26V2/ = (^ - + (52/2/). 

Hence, the results hold. 

(ii) Since the Vciriance and MSE of g aire functions of 6, it is desirable to to transform 
g into variance-stabilizing estimators. The variance of g can be decomposed as = a^i^b^S^. 

5 
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Following Laubscher (1959), the reported variance-stabilizing and normalizing transforma- 
tion of g is obtained. 

(iii) The results are trivial since, by definition, Bias{g) ^ + and MSE{g) - (tI - 
[Bias{g)f. Analogous arguments apply to the derivation of .Bia5(^) and MSE{g). \\ 

Novick and Jackson (1974) suggested another approximation for c/, namely, cf = 
x/1 - (3/2/) with an error about .01 for / > 5. Under this approach. Jig - [y/2f/{2f - 3)]5 
and pLg = [^2//(2/- i)]6. The approximation given by Hedges (1981) is preferred since it 
is more precise and yields a smaller biets in the estimation of 6. In the limit, cj^ reduces 
to 1 under both approaches. When f is Ets small as 3, cj^ is equal to 1.38, 1.41 and 1.37 
according to the exact formula and those given by Novick and Jackson (1974) and Hedges 
(1981), respectively. 

Corollary 2 (Unbiased estimators). Consider the estimators h = cjg and h = c^g. Then 
h = h and the following results can be applied to both h and h: 

(i) The distribution of h is kcft^f^k-^e) with mean fXh =^6 and its variance can be specified 
as, 

(a) (Exact) ch^ = [c}{ji^){P -f 6^)] - 6\ 

(b) (Approximate) Bl = [(^)(fc2 -f S^)] - 6\ 

(c) (Asymptotic) <oo = 4oo' 

(ii) A variance-stabilizing and normalizing transformation of h is: 

ft* = 6^^5inft~^(6/ia^^ft) = fc^^in|6/ift + cr;^|, 

where an = c/a, hh = c/6 and, 6^^[sznft-^(6/ia^^ft) - $inh'^{hhal^6)] % iV(0, 1). 

(iii) The bias of h is 0 and its mean-square error is equal to a\ 

Proof: The unbiased estimators h and ft have the same distributions because it can 
be written that, 

ft = cyg = [s/f/j]cjg = [s/f/fyAy/ITng = cjg = h. 

The remaining results can be derived due to the proof of Corrolarv 1 as well as the 
fact that a\ = c^cr^ and c/ is increased to 1 as f increases to infinity. Therefore, the limiting 
distributions of h and g are the same by Slutsky's theorem (Serfling, 1980). || 

An unbiased estimate of can be expressed in the form of Sh" = [c/^(j^)(*^ + /i^)]-/i^- 
Hedges and Olkin (1985) proposed a variance- stabilizing and normalizing transformation 

6 
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for the estimator h2 = cj^(X -Y)/S2 as: 

hi = V2Nsinh'^[h2/q] = V2N^Jln\{h2/q) + {hl/q^) + ll 

where q = ^4 -f 2(n/m) 2(m/n) eind iV = ti4 m. This equation is derived from the asymptotic 
distribution of /12. Kraemer (1983) suggested another trcinsformation method from the 
relationship between the distributions of h and the product- moment correlation coefficient. 
Both the Laubscherian trcinsformation cind that of Kraemer (1983) are based on the exact 
distributions ofhr and thus expected to be more accurate for small sample cases than the 
method of Hedges and Olkin (1985). In practice, the Laubscherian equation is easier to 
use than Kraemer's (1983) procedure since the latter may require cin additional step of 
converting the transformed values in terms of correlation coefficients to the original scale 
of measurement. 

The efficiency of estimators are often evaluated according to the minimum mean- 
square error criterion. In the following, two additional estimators are derived from proce* 
dures that serve this purpose. 

Proposition 2 . (Shrunken estimators) 

(i) (Thompson, 1968). Let d be cin estimator of 6. The minimization of MSE{wTd) - 

results in the weight wt of the form: 

(ii) (Hedges and Olkin, 1985). Consider an estimator d of 5 such that — ajj ^h<^^' 
Then the MSE of any linear transformation of d, say d^ =^ twi/d + g^; where tw^ and 
qf{ are 

some constants; is minimized by defining the weight as. 
Proof : 

(i) The given expression for tuy that minimizes the mean-square error (MSE) of dj = tujd, 

MSE{wTd) = e{wTd - - -f /id) + Md - 2tuxMdi 

can be obtained ets a solution the partial derivative of 

bMSE{wTd)/owT = 2wt{(tI - /x^) - ipi = 0. 
7 
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(ii) See Hedges and Olkin (1985), pp. 105-106. || 

Some applications of the results in ProvostUon & axe now explored. The distribution of 
d belongs to the family of non-central t distribution with degree of freedom f. When f is 
large, the distribution of d is approTdmately normal (Johnson and Welch, 1939) with mean 
and variance of 6 zmd k"'^ + (5^/2/), respectively { Corollaries 1 and 2 ). Since the theoretical 
origin of the normal distribution is zero, following Thompson (1968), a shrunken estimator 
of 6 can be formed as: 

or, alternatively, dr = [{l/k^S^) + (1/2/) + 1J"^6. 

As another application of Thompson's result (1968), let d = h then a minimum MSE 
estimator {d^) can be expressed as: 

dr = = +f^l)h = ( [//(/-2)][fc^ + g2] )^- 

An unbiased estimate of loy, denoted as Wy, is obtained upon replacing 6 by h. The 
corresponding shrunken estimator in terms of the moments of h under Hedges and Olkin's 
(1985) procedure is of the form: 

So fax, five generalized estimators of effect size, namely g^g^h^dr and ci^ have been 
discussed. Note that these estimators have the same signs. They are now compared 
according to the following three criteria: the size of their absolute values, biasedness and 
MSE magnitudes. The comparisons cire performed on the basis that all relevant estimators 
are computed with the scime sample sizes. 

Provosiiion S (Comparing the estimators). Given that / > 2 and f > the relative 
rank-ordering magnitudes of g, d^, and dj on the basis of thdr absolute values, degrees 
of bias and MSE magnitudes, respectively, are: 

(i) \g\>\9\>\h\>\dH\>\dTU 

(ii) \Bias{dT)\ > \Bias{g)\ > \Bias{g)\ > \Bia8{dH)l 

(iii) MSE{g) > MSE{g) > MSE{h) > MSEidn) > MSEidx). 
Proof : 

8 
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(i) This result is proved by means of pairwise comparisons among the estimators of 6, 



starting with the claimed smallest estimatorb, r-\raely dj and d//. Since dr ^ [fi^/ik^ -t h^)]dH 
and [h^/{k^ + h^)] < 1, we have {drl < Idnl Now, dn ~ iDnh implies that \h\ > Idnl iiy^H <i or 
(/ - 2)// < cj^. But this condition is tenable since cj^ > 1 and (/ ~ 2)// < 1. Next, h = cjg 
indicates that \h\ < \g\ since c/ < 1. Finally, \g\ = \{Jf/f)g\ > \g\ when / > /. 




(ii) With Bias(h) 2=, 0 already obtained, the pairwise compcirisons is conducted first 



\Bias{g)\ > \Bias{g)\ because ix§ > fXg when f > f. Fbr41y, \Bias{dT)\ > \Bias{g)\] where 



(iii) Consider MSE{dT) = e{dT - A^rfr)^ = ^{'^Tdn - ^TfJ^dnf = VT^MSE{dH)< 
Then, MSEidn) > MSE{dT) because vt < 1. Analogously, MSE{dH) = wjje{h'6f = w%(rl 
is less than MSE{h) = because tw// < 1. Next, MSE{g) = + (Bias(ff))2 is larger than 
MSE{h) = cjcr^ since c) < i and {Bias{g))^ > 0. Finally, M5E(^) ^ e{g - a*^)^ = {f/fH9 - P^g^) = 
{f/f)MSE{g) is larger than MSE{g) when /> /. || 

For a given value of f and /, the biased values for each of the five estimators under 
consideration are smaller than its relevant MSB values. As f increases, the biased values 
of these estimators would rapidly converge to zero when their expected values approach 6. 
In the limit, the corresponding mean-square errors (MSE) would not always converge to 
zero and when they do, not as rapidly. In pcirticular, the mean square errors of g, g, h and 
dn would be equal to o-^ which are expected to be Isigei than since the latter is of 
the form v^^wla^^^ where vt and w^i are less than one. Note that the results in Provosxixon S 
can applied to all cases listed in Table 1 since /, > /t (i = 1, 5), except that /o < /o when 
n > m. The exception will be illustrated in a Monte-Ccirlo study. 

The findings above cam be generalized to other estimators of 6. In general, bias and 
I. 3E values grow with the increase of 6 and reduce as the associated degreeb ^f freedom 
increase. Values of Bias(g) and Bias(g) share the same signs vrith 5 whereas those of 
Bias(<i^) and Bias{dT) have the oppisite signs to 6. From the results in Proposition S j it 
is clear that the popular estimator g, proposed by Glass (1976), tends to overestimate 
the effect size {6). According to the criteria of minimum bias and mean square error, the 

9 
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estimators h, dn and should be preferred to g and g. Moreover, is more favorable to 
djf on the basis of the variance stabilization criterion. In terms of their absolute values, h 
is unbiased whereas dj would tend to underestimate 6 . ThereJjre, both h and dj should 
be computed in a raeta-analj^sis study. Tt*'. former is used in the analj^sis of effect size 
of the experiment under consideration and the latter, as an indication of its lower bound. 

Properties of Some Estimators of Effect Size 

The distributions of some specific estimators of effect size {S) are now studied with 
respect to the general properties presented above. In particular, some specific forms of 
the five estimators g%j9t,thtdH, and dx< are exaumned (The subscript i represents the case in 
which 5,, listed in Table 1 , is used in the expression of the estimator under consideration). 
According to ProvosiUon the bictsed estimators (5, and g,,i = 1, 4) eire relatively less 
effective than the unbiased and shrunken estimators (/^, d//, and dy,, i = 1, 4) in the 
estimation of effect sizes. 

Estimators under variance homogeneity 

The main properties of estimators listed in Table 1 for the case of = can be 
sunmiarized as follows: 

Corollary $ (Variance homogeneity) 

(i) (Bictsed estimators) 

(a) \9i\ >\9i\> I52I > \92\ if 5^ > Si and [52! > I52I > Iffi! > \9i\ if > 5^ when n = m. 

(b) \Bias{gi)\ > \Bia5f9,)\ > \Bias{92)\ > \Bias{g2)l 

(c) MSE{gi) > MSE{gi) > MSE{g2) > MSEigz). 

(ii) (Unbiased estimators) 

(a) \h2\ > \hi\ if Si. > 5^ when n = m whereas \hi\ > I/12I if 5^ > S^. 

(b) MSE{hi) > MSE{h2) 

(iii) (Shrunken estimators) 

(a) \dTi\ > \dT2\ if \hi\ > |/i2l and vice versa if \hi\ < |/i2|. 

(b) \Bias{dT2)\ > \Bias{dTi) 

10 
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(c) MSE{dT,)>MSE{dT,) 
Proof . 

(i) (Biased estimators) 

(a) The results in Proposition 3 implies that \g^\ > \gx\ (for i = 1, 2). From the definition 
of Sf and Sf in Tables 1, it only requires to find the condition for 5| > Si so that \gi\ > |o2|. 
Since 5| - 5? = ^^'^fflj^'z')^^^'^ - 5^ = the resulting condition is as stated in this 
Corollctry . Analogously, it is necessary to show that Si > 5^ to obtain \g2\ > Since 
the condition §1 > 5| implies that (n-f m - 2)(m - 1)5^ > m[{n - 1)5^ + (m - 1)5^] or, upon 
simplification, 5?- > [m{n-l)/n{m'- the result is obtained. 

(b) and (c) From Proposition S j it can be seen that J5ias(§») > Bias{gt) axid MSE{gx) > 
MSE{g^)^ for all i. Moreover, since biased and MSE values are monotonic decreasing 
functions of and from the fact that /2 > fi ( Table l ), we have 5ias{gi) > Bias{g2) and 
MSE[gi) > MSE{g2)> Therefore, the results hold. 

(ii) (Unbiased estimators) 

(a) The condition for A2 > hi can be readily specified from the facts that A2 - Ai = 
- Cf^giy Cf^ > c/j (because c/. is a monotonic increasing function of /2 > /13 aJid 

1^2! > \9i\ for the case in which S^r > S^ when n = m). On the other hand, to find the 
condition for Ai > A25 a simple proof can be carried out by using the approximation 
(Novick and Jackson 's,1974). The resulting forms of approximate unbiased estimators are 
expressed as ^1 = [^(2m-5)/2 EatK^-^) and A2 = [i/(27i - 7)/2(£x + Ey )](^'- ^^ere Y.x 
and Ey are defined ^^x = YLi^ - '^f and Ey = E (^^ " ^)^ respectively. Then, from the 
difference 

h\-h\= {(2m - -2(m - 1) J^)}, 

one can conclude that A| > hi if 5^ > [2(m- l)/(2m+ 5)]5^. It is clear that - A? > 0 if 
(2m- 5)(53;f -2(m- 1) J^y) > 0. Since 2(m-l)/(2m+5) is less than one, the condition S\ > 5^ 
would yield the result hi > A2. 

(b) For the unbiased estimators hi and Ay, we have 

MSE{hi)-MSE(hi) = al-al.. 

Therefore, the result holds since a\ is a monotonic decreasing function of f and the fact 
that h> h. 

11 
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(iii) (Shrunken estimators) 

(a) Consider dr, = wm[hV{k^^ + h^)]hi for i = 1, 2; where wm = cJ^^K/i - 2)//,]. By 
applying the approximation c^. (Novick and Jackson, 1974), it can be written that WHt = 
[(/,/(/»-!. 5)][(/»-2)//,] % 1 for ft > 3. Hence, the magnitude and direction of dy^ are essentially 
determined by hi. 

(b) Without loss of generality, assume 6 > 0. The bias of dxi can b^* expressed as 
Bias{dT^) = WHx{[6^/{k^ + 6^)] - 1}6 = -whxP6/{P + 6^). Now, Bias{dT2) - J5za5(dri) = (t^^^i - 
WH2)k^6{k'^ + > 0 since /2 > fi emd wm is a monotonic decrecising function of /». 

(c) Since MSE{dTt) is a monotonic decreaising function of the result follows for 
/2</i. II 

The results in Corollary 3 imply that, with the exception of the shrunken estimators, 
the pooled varicinces (used in the computation of g2i92if^2 and ir,) are relatively more 
efficient thcin the unpooled vajicinces ^ *elding estimators with smaller biased and MSE 
vsdues. The Scime observation applies to the following ca^e of heterogeneous variances with 
known varicince ratios. 

Estimators under variance heterogeneity with known ratios 
Corollary i {a\ (7^, assuming r = f^x/^r) 

(i) (Biased estimators) 

(a) \h\ > \gz\ > M > Iff4| if 5^ > [r(n+ ri\r+ l)/(n - 1)]5^ and M > bl > l^al > bl if 

(b) \Bias{g3)\ > \Bias{g2)\ > \Bias{g^)\ > |J5£as(g4)|- 

(c) MSEih) > MSE{g2) > MSE{g^) > MSE{g^). 

(ii) (UnbizLsed estimators) 

(a) \h^\ > Iftal if r > 1 or 5?. > 5^ and {h^l > \h^\ if r < 1. 

(b) MSE{h3)>MSE{h^). 

(iii) (Shrunken estimators) 

(a) \dT3\ > |dr4| if jfcal > 1^41 and vice versa if jfeal < \h^\* 

(b) |J5ta5(dr4)| > \Bia${dT3) 

(c) MSE{dT3)>MSE{dTA) 
Proof . 

12 
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(T) (Biased estimators) 

(a) Since 5| - S\ = - 2)(r + 1) - 2r]/2(i\r - 2)};^:^ - 2)-^];[:;^ the condition for 
Si > Si so that 1^41 > \h\ is S^ > {2(n - l)/(m - 1)[{N - 2)(r + 1) - 2r]}5^ or > 5^ as stated 
above. Similarly, since Sl^Sl^ [T,xl^'^\ + [^"^ - + l)]Er » the condition for 5| > 5| 
so that bl > \h\ is 5^ > [r(i\rr + n + l)/(n- 1)]55^. 

(b) and (c) The proofs for the order-ranking bias and mean squcire error values require 
the same cirguments as present'*'^ ^or the biased estimators in Corollary L 

(ii) (Unbiased estimators) 

(a) Without loss of generality, the proof can be simplified by means of the approxima- 
tion (Novick and Jackson, 1974). The corresponding approximate forms for the unbiased 
estimators and are ^3 = >/(2m- 5)/2(m- l)(r-y)/53, and ^4 = V{2N -7)/2{N - 2)(X - 
Y)/S^, respectively. Consider the difference U^H = {[^^^^0^]^\^l^}{X-Y^ = C34/P34; 
where C734 = [(1 - r){2nm -f 2m + 9m)] + 2n(r - 4) -f (13 - 7r) and P34 = 2(n -f m - 2)(r -f l)(m - 1). 
Since P34 > 0, the sign of hi - hi is the Scime 2ls that of C34. The results hold since C34 < 0 
if r > 1 and C34> 0 if r < 1. 

(b) and (c) The proofs are trivial, following the same arguments used in Corrolarv 3 
with respect to the unbiaised estimators. 

(iii) (Shrunken estimators) 

The proof follows the scime cirguments as given in Coronary i upon replacing dri, and 
dT2 by dra and dT4, respectively. || 

The results so far imply that estimators /12 and dr^ must be preferred to h^ and drj. 
The choice betwen hz and ^4 should depend on whether r is larger than one or not. 

Estimators under the Behrens*Fisher condition 

Corollary 5 (Bi2Lsed estimators): The distributions of g$ = (X -Y)IS^^ g^^p^ - Y)/Ssy 
and /15 =c/,ff5, respectively, are fci(/„jfc-i^), and Cf^kt^j^^k-H) where: 

(b) k = {hlh), and 

(c) /s and /s are specified in Table 1. 

Proof : The distributions of gs, and /15 can be derived from the results of Provositton 1 . 
The expressions for k, /s and /s are obtained as follows: 

13 
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(a) The effect size suggested by Cohen (1977) when (^x^a^ is of the form 6 = {fix - 
/iy)/o- where a =. -r o-y)/25 a root-mean-square value (Cohen, 1977, p.44). Hence, from 

Provosiiion 1 ^ the expression for k is obtained. 

(b) An unbiased estimate of is Si = (5^ + 5^)/2 = + • Then, 
S?MayU)Xfj^^ where 

{c4/4(n - 1)} + {4/4(m - 1)} {ay{n - 1)} + {c7^/(m - 1)} ' 

according to Welch's (1938) procedure. Upon simplification, the expression of /s is given 
as 

(m-l)aj^ + (Ti-l)c7^ ' 

In Table 1, an unbiased estimates of aie obtained upon replacing and o-^ by Sx and 
5^ , respectively, 

(c) The MLE of under the Behrens-Fisher condition can be expressed as Si = 
{Sx + 5y}/2 = ([(^-^ )H^y + ((^-^)H^r} . 

Then, ^"^•^^>^(--M+^"''/"^^^f--M ^ Hence, from Welch's (1938) procedure, one gets: 

f _ [{n- l)<Tx/n+ (m- l)<T^/mf 
- {(^ _ l)aj^/7i2} + {(m - l)a^/m2}' 

The above expression cciii be simplified £is 

An unbi2Lsed estimate of /s are given in Table 1, || 

By applying the results in Corollary moments of the exact, approximate and lim- 
iting distributions of ^5 and ^5 can be derived. The expected values and variances of 
and ^5 are equal to those computed for g and ^ [ Corollary l ) when / is defined as /s or /s, 
respectively, Similcirly, it is possible to obtain moments for the unbiased estimators (/15) 
and the corresponding shrunken estimators (drs) by mezins of Corollary g and Provosiiion 2 ^ 
respectively. As will be seen in the following simulation study, /s > /s when m > n. Given 
this condition, the relative magnitudes, biased and MSE values of gs^hif^^^djj^. and dr^ 
follow the same patterns presented in ProposUion 3 . For the case in which /s < /s (when 
m < n), the relationships expressed in Provosiiion 3 still hold upon letting g and g sv^itch 

14 
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places. In other words, the rankings among the unbiased and shrunken estimators are not 
affected by the relative sizes of n and m since they are computed on the basis of K (which 
is equal to hi for all fi and f{), 

Monte-Carlo Results 

Comparisons of Estimators of Effect Size Under Balanced Designs 

Properties of the five estimators g,g,h,dH and were studied by means of a Monte- 
Caxlo method under the balanced designs (n = m) as well as under several configurations of 
saiiiple sizes (n and m) and variance ratios (r) for the three cases presented in Table 1 . Under 
the bcilanced designs, the Scime computational expressions for the above five estimators are 
found in both Case 1 (Variance Homogeneity) and Case 3 (The Behrens-Fisher problem) 
as implied by the following results: 

Corollary 6 . (Balanced designs). Let f = Sx/Sy represent an unbiased estimator of 
^x/^Y' When n = m, the following properties are observed: 

(a) k = y/2/n in all three cases presented in Table L 

(b) 5| = 5|, 5| = Sh Si = SWif + l)/2 and 5| = V(5^/2f) 4- (5^/2). 

(c) /i = /3 = (n- 1), A = /3 = n, /2 = /4 = 2(n-l), A = A = 2n, /s = /s = (n- 1)(1 + f)V(l + r2). 
Proof : 

(a) In Case 1 (Vciriance Homogeneity) k — y/{n'^m)/nm = y/2/n for n = m. In Case 2 
(r = cr^/cr^), k = y/2{mr -f n)/nm{r + 1) = y2n(r-f l)/n2(r-f 1) = y/2/n. Similarly, in Case 

(b) and (c) By setting n = m in the expressions for 5?, Sf, fi and fi (i= 1, ...,5), 
the results arc obtained upon simplifying similar terms. || 

Vcilues of the five estimators of effect size mentioned above were generated with 
increasing sample sizes (5(1)105) and increasing effect sizes (0(.2)2). In Case 1 {a\ = crj) 
and Case 2 (r = cr^ /^y )? the degrees of freedom /i and fx were used in the estimation of the 
moments for g and ^, respectively; whereas the degree of freedom /z was used in association 
vdth the remaining three estimators (h, dn and dr)- In Case 3 (a^ ^ a^), since /s = /s, the 
degree of freedom /s were used in computing the moments of all five estimators. Under 
Cases 2 and 3, the condition of variance heterogeneity was specified by setting the values 
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of fas 2(1)10. 

For a given value of effect size, the distributions of th^ mecins cind variances of the five 
estimators and dj) in any configurations of sample sizes under consideration can 

be typically represented by the plots in Fioure L The corresponding biased emd MSE values 
are generally performed as the distributions plotted in Fioure 2 . For 5 > 0, the following 
relationships are observed across all simulated configurations of sample sizes (n m), 
effect sizes {S) and variajice conditions (r): 

Insert Figures 1 and 2 about here 

(a) fi{g) > fi{g) > fi{h) > ^{dn) > /x(^), and 

(b) crl>al>al>al^>al. 

(c) The relative magnitudes of biased emd MSE values for the five estimators followed 
the patterns described in ProvosHion S . 

The above relationships car; also be applied to the case ol negative effect size provided 
that values of the mecins in (a) are expressed in absolute terms. As sample sizes increase, 
the main properties of ctr are: (i) minimum variance, (ii) stability (values of ^r^^ are quite 
small even for small degrees of freedom), and (iii) minimum mean square error, despite 
the large absolute values of Buis(dr). 

The typical behaviors of the five estimators of effect size under consideration with 
respect to the constraint of fixed degrees of freedom and varied effect sizes 2tre depicted in 
Figure 3 and Ficure J . 

Insert Figures 3 cind 4 about here 

In general, the moments of the estimators are monotonic increasing functions of 6. 
The notable exceptions are Btas{h)^ which is always equal to zero, Bias{dT) for 6 around zero, 
and Bias{dH)> As 6 increases, the means of the estimators grow much faster than other 
moments. In this case, the main properties of dr can be summarized as: (i) minimum 
variance, (ii) minimum mecin square error and (iii) largest bias. For 5 > 0, as effect size 
becomes larger, the absolute value of Bia3{dT) is reduced toward zero. On the contrary, 
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> 

the same absolute values of the other estimators depart further from zero. In short, the 
shrunken estimator dx is most efficient, as compared to g, h and d^, in the estimation 
of effect size when the Scimple sizes are small and 6 is large. 

General Comparisons of Estimators of Effect Size 

The specifications of the simulation parameters (n, m, r, f and /) used to generate 
values of the five estimators in several combinations of sample sizes and variance conditions 
are reported in Table 2. In each combination of f, f (or /) and vatriance conditions (denoted 

Insert Table 2 about here 



as Case 1, Case 2 and Case 3), six data sets containing values of the five estimators 
were generated, corresponding to the six configurations of sam^^le sizes (n and m) (de- 
noted as ID in Table 2). Under Case 1 (a^ = a^) and Case 2 (cr^/crf. = r), the de- 
grees of freedom fi and /i were used to derive the moments of g and ^, respectively; 
whereas moments of the estimators h, and dr were computed on the basis of /2* 
Similarly, under Case 3 (cr^ cr?^), properties of g were derived basing on /s and mo- 
ments of the other four estimators were computed by means of /s. To generate data 
under Case 3 (cr^ cr^), it is assumed that f is an unbiased estimate of r where f = 
Sj^/S^. Therefore, the estimates of /s and /s in Table 1 are revised respectively as: 

fr = 

(»n-l)f?+(n-l) 

f _ fyn(n-l)f+n(yn-l)1^ 
is — ni^(n- l)f 2+n2 (m- 1) 

As shown in Table 2, /s > /s when m > n and the reverse is true when m < n. The 
resulting expected values of the means, variances, bias and mean-square errors of the five 
estimators over all configurations under study are reported in Tables 3 to 8. 



Insert Tables 3 to 8 about here 
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On the basis of simulated data, principal characteristics of the distributions of the 
five estimators under consideration can be summarized as follows: 

(a) When 5 = 0, all estimates of effect size are unbiased. Similarly, when 5 = 1, the biased 
values are very small. 

(b) Expected values of the means, variances, bias and mean square errors of estimators of 
effect size are monotonic increasing functions of 6. Except /x(dr), the above moments 
are monotonic decreasing functions of scimple sizes* 

(c) As 6 increases, expected values of the means grow faster than expected values of other 
parameters, with fi{g) and fi{g) increase fastest. The values of MSE(g) and MSE{g) 
increase more rapidly than mean squcire errors of the other estimators. While the 
biased values of other estimators increase in absolute terms, Bia${dH) and Bias{dT) 
reduce to zero. 

(d) As sample size increases, the biased values reduce faster than other parameters. The 
magnitude of reduction (R) can be arranged in descending order as R(Bias), R(MSE), 
R{a^) and R{fi). 

(e) Within a given configuration of 6, sample sizes (n and m), and variance condition 
(r), values of MSE are substantially larger than those of Bias. Moreover, the rela- 
tive magnitudes of Bias and MSE values for the*five;:estima.tors in..eacll^Bimtflated a 
configuration conform to the relationships described in Proposition 

(f) In Case 2 (^^/^y = as r increases moments of the estimators of effect size change 
very little in most configurations (ID). In Case 3, ^th the exception of 
which tends to stabilize, variances of the other parameters generally increase (reduce) 
for small (large) degrees of freedom. 

In the presence of the Behrens-Fisher condition, vziriances of the estimators of effect 
size were inflated if the degrees of freedom under the assumption of variance homogeneity 
were used. However, this adverse effect is minimal with respect to the shrunken estimator 
dr. For example, for the configuration of 5 = 1 and ID = 1, the values ofa^ axe .97 in Case 
1, .44 (when r = 2) and .65 (when r = 10) in Ca^e 3. The corresponding values ola^^ axe 
.21, .20 and .22. In general, as r increases, moments of the estimators increase for small 
degrees of freedom (ID from 1 to 3) and reduce when the degrees of freedom are large (ID 
from 4 to 6). In all cases, the values of a^^ are small, ranging from zero to .22, and quite 
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stabilized as r increases. 



Conclusions 



The Monte-Carlo results tend to support the previous recommendation, based on 
theoretical analysis, that the estimators h and dr should be used in conducting a meta- 
analytic study. The use of both estimators will ensure the properties of unbiasedness and 
minimum MSE in the estimation of effect size (5), Moreover, on the basis of simulated 
data, values of both h and dr were not affected as severely as g and g in the presence of 
violations to the assumption of variance homogeneity. The moments of dr were quite stable 
across the different variance conditions under consideration. Clearly, due to its properties 
of minimum MSE and variance stability, dr perform better than Glass-type estimators (g, 
g) and Hedges- type estimators (h and djj) when sample sizes are small and effect size is 
large In general, balanced designs should be attempted whenever possible, to minimize the 
effects of variance inflation caused by violations to the variance homogeneity assumption. 
Further research is needed to assess the relative effectiveness of estimators of 6 in the 
context of statistical inferences and data analysis. 
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Table 1 

w jme population variances (cr^), noncentrality parameters (A) and estimators of cr. 



Case 1: Variance homogeneity 

(a) Population vari^mces: cr = crx = c^r- 

(b) Noncentral parameter: A - k' H = $(}rinm/{n + m)6 = n"^/^6. 

(c) Estimators of a: 

Si =5r5 

^2 - V (n+ni-2) ^ 

51 = Sy = ^/(m- l)55./m, 

52 = \/[n^ +m5f]/(n+ m) where 5^ = [(n - l)/n]5^ and S^r = [{m - l)/m]S^r. 

(d) Degrees of freedom of the chi-square distributions for S^/a^ and S^a^^ i = 1, 2; re- 
spectively: 

/i=(m-l), 

/2=(n+m-2), 

fi =m, 

A =(n +m). 



Case 2: Varicuice heterogeneity 
(with cr^/cTy- = r, where r is a known constant) 

(a) Population variances: cr^ = -f r)/2 

(b) Noncentral parameter: A = k~^6 = {>/nm(r + l)/2(mr + n)}6 

(c) Estimators of cr: 

53=5yV(r+l)/2 

Q _ / [(n-l)5j[./rl+(m-l)5?. 
•^4 - y (n+m-2) ^ 

Sz = S3y/{in-l)/m^ 

54 = ^J[{n/r)S^ + mS^]/{n + m). 

(d) Degrees of freedom in the chi-square distributions for fxS^/a^ and fiS^cr^^ i = 3, 4; 
respectively: /a = /i, /4 = /2, /a = A and h = A- 
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Case 3: The Behrens-Pisher problem (cr^^crf ) 



(a) Population VciricLnces: a = ^(cr^ + cr^)/2 



(b) Noncentral parameter: A = Jb-^fi = y^i^^^ 

(c) Estimators of a: 

g5 = V (5^-f 5?0/2 
55 = \/[5i + 5^1/2, 

(d) Estimates cf degrees of freedom of the chi-square distributions for Sl/a^ and Sl/a^] 
respectively: 

-'^ " (m-l)S5.+(n-l)S* 
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Typical values of means and variances for five 
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Typical biased and mean square error values of 
five estimators of effect size (g, g, h, d.., d^) 
under balanced design with ^=-6&n=m=5,.-- 
105. 
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Figure 3 

Typical values of means and variances for five 
estimators of effect size (g, g, h, d^^, d^) under 
balanced design with n = m ^ 30 and ^ = 0, 2- 
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Figure 4 

Typical biased and mean square error values of five 

estimators of effect size (g, g, h, d„, d„) under 
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balanced design with n = m = 30 and S - 0, 2. 
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Table 2 



ID 



Notes: 
(a) 
(b) 



6 
6 
6 

51 
51 
51 



6 
21 
51 

6 
21 
51 



Cases 1 and 2 
r = 1, 2, 10 



5 
20 
50 

5 
20 
50 



6 
21 
51 

6 
21 
51 



10 

25 
55 

55 
70 
100 





Case 

•'2 


3 


r = 


10 




^5 




t 


5 




9.00 
10.59 
10.98 


9.00 
1 1.42 
12.19 


5 
6 
6 


.99 
.03 
.04 


5 
6 
6 


.99 
.19 

24 


32. 14 
69.23 
90.00 


36. 18 
69.42 
90.00 


55 
59 
59 


.00 
.02 
.90 


54 

58. 

59. 


90 
80 
90 



r = An unbiased estimate of the variance ratio (r =62/^2). 
in Case . (Variance Homogeneity, r - 1), the degrees of fre^edom and \ 
were used in the computation of the parameters for g anC respectively. 
The degree of freedom fgvere ued to derive values of the parameters for 
h, ^ and dp. The same procedure was applied in Cae 2 (Variance 
Heterogeneity with known r)) since f3 - t^, 1 ^ . ^ . f j„ ^ase 3 

(The Dehrens-Fisher condition. «2 . g2,, ^he degree of freedom was used 
in association with ^ whereas the parameters of the other four estimators 
were obtained on the basis of £5. , 
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Table 3 



Means of some generalized estimators of effect size (S) for six configurations of sample sizes (id) 







Case 1 


Case 2 


Case 3 




ID 




r = 


1 






r « 


2 






r = 


10 






r = 


2 






r = 


10 










iJ 

o 


o 




1 






ti 


X 


^g 




H 


^ d 

I 


6 






^d 


g 


g 


p. 


\ 


0 
0 
0 


1 

2 
3 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00' 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0 . 00 


0.00 
0.00 
0 . 00 


0.00 
0.00 
0 .00 


0.00 
0.00 
n nn 


0.00 
0.00 

n nn 


0.00 
0.00 
n nn 


0 
0 
0 


.00 
.00 
.00 


0 

U 
0 


4 

e 
0 

6 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0*00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
O.OD 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0.00 
0.00 
0.00 


0 
u 
0 


.00 
.00 
.00 


.6 
.6 
.6 


2 
3 


0.71 
0.62 
0.61 


0.78 
0.64 
0.61 


0.56 
0.59 
0.59 


0.29 
0.37 
0.39 


0.71 
0.62 
0.61 


0.79 
0.64 
0.61 


0.56 
0.59 
0.59 


0.29 
0.34 
0.36 


0.71 
0.62 
0.61 


0.78 
0.64 
0.61 


0.56 
0.59 
0.59 


0.29 
0.31 
0.32 


0.66 
0.65 
0.64 


0.66 
0.67 
0.68 


0.56 
0.57 
0.57 


0.31 
0.34 
0.35 


0.69 
0.69 
0.69 


0.69 
0.70 
0.70 


0.53 
0.53 
0.53 


0 
0 
0 


.28 
.29 
.29 


.6 

.6 
.6 


4 
5 
6 


0.71 
0.62 
0.61 


0.78 
0.64 
0.61 


0.59 
0.60 
0.60 


0.39 
0.50 
0.54 


0.71 
0.62 
0.61 
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0.64 
0.61 


0.59 
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0.60 
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0.62 
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0.53 
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0.61 
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0.59 
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0 
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.52 
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1.01 
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1.04 
1.01 
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1.06 
1.02 


0.94 
0.98 
0.99 
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1.04 
1.01 
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1.02 
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0.98 
0.99 
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0.94 
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0.76 
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1.17 
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0.99 
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1 .06 
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0.99 
0.99 
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0.99 
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0.96 
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0.99 
0.99 
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0.95 
0.96 
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1.01 
1.01 


1.01 
1.01 
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0.99 
0.99 
0.99 


0 
0 

0 


94 
95 
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Table 4 



Variances of some generalized estimators o£ effect Size (i) for six configurations 
of sample sizes (ID) 
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Table 5 



Variances of some generalized estimators of effect size ii) for six 
configurations of sample sizes (ID) under Case 3 (The Dehrens-Fi sher 
problem) 
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Table 6 

Biased values of some generalized estimators of effect size (i) for six configurations of sample sizes (ID) 
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-0.07 
-0.06 


0.01 
0.01 
0.00 


0.05 
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Table 7 



Mean square errors (mSE) oi some generalized estimators oi effect size ii) for 
six configurations of sample sizes (ID) 
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Table 8 

Mean square errors (mSE) ot some generalized estimators ot etCect size (i) for 
six configurations of sample sizes (id) under Case 3 (The Dehrcns-Fisher problem) 
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